A Comparative Study of Measures to Evaluate Medical Students’ Performances

Purpose. To assess how new National Board of Medical Examiners (NBME) performance examinations—computerbased case simulations (CBX) and standardized patient exams (SPX)—compare with each other and with traditional internal and external measures of medical students’ performances. Secondary objectives examined attitudes of students toward new and traditional evaluation modalities. Method. Fourth-year students (n = 155) at the University of California, Los Angeles, School of Medicine (including joint programs at Charles R. Drew University of Medicine and Science and University of California, Riverside) were assigned two days of performance examinations (eight SPXs, ten CBXs, and a self-administered attitudinal survey). The CBX was scored by the NBME and the SPX by a NBME/Macy consortium. Scores were linked to the survey and correlated with archival student data, including traditional performance indicators (licensing board scores, grade-point averages, etc.). Results. Of the 155 students, 95% completed the testing. The CBX and the SPX had low to moderate statistically significant correlations with each other and with traditional measures of performance. Traditional measures were intercorrelated at higher levels than with the CBX or SPX. Students’ perceptions of the various evaluation methods varied based on the assessment. These findings are consistent with the theoretical construct for development of performance examinations. For example, to assess clinical decision making, students rated the CBX best, while they rated multiple-choice examinations best to assess knowledge. Conclusion. Examination results and student perception studies provide converging evidence that performance examinations measure different physician competency domains and support using multipronged assessment approaches. Acad. Med. 2000;75:825–833. Accurately measuring physicians’ competencies is a crucial step toward imDr. Edelstein is senior associate dean of academic affairs and associate professor, Department of Family Medicine, Charles R. Drew University of Medicine and Science. Ms. Reid was senior statistician, Department of Psychiatry and Biobehavioral Sciences, UCLA School of Medicine. Dr. Usatine is associate professor, Department of Family Medicine, and assistant dean of student affairs, UCLA School of Medicine. Dr. Wilkes is associate professor, UCLA Office of the Dean, Center for Educational Development and Research, and the Division of General Internal Medicine and Health Services Research. Correspondence and requests for reprints should be addressed to Dr. Edelstein, Charles R. Drew University of Medicine and Science, College of Medicine —Academic Affairs, 1731 E. 120th Street, Los Angeles, CA 90059; e-mail: ^[email protected]&. proving medical education and, in turn, improving health care. The need to rigorously evaluate physicians’ performances (by measuring, among other things, process, outcome, and effectiveness) pervades the economic, social, and scientific debates surrounding health care reform. The growing concern for accountability is demonstrated in current proposals in both the public and private sectors to develop and implement practice guidelines, outcomes assessment, evidence-based medicine, and measures such as consumer satisfaction surveys. Professional oversight bodies at the national, state, and local levels are challenging medical schools to undertake curricular reforms, including assessment procedures that will assure that graduates will be able to practice successfully and contribute to improved patient care in the newly restructured health care environment in the United States. Given the pronounced scrutiny of physicians’ performances and educational reform, it is easy to understand why new evaluation methods are receiving increased attention in medical schools. There is concern that traditional measures of performance (multiple-choice examinations, medical licensing examinations, grades, and narrative rating forms) are constrained,